Practical Software Quality 

A guide in progress 


Presented at 2015 Flight Software Workshop by co-authors: 
Dan Painter -NASA IV&V 
Matt Rhodes - MathWorks 



What's your [mis]fortune cookie say? 




What is quality? 


"The standard of something as measured against 
other things of a similar kind; the degree of excellence of 
something." 

Oxford English Dictionary 


What is quality? 


"The Standard of something as measured against 
other things of a similar kind; the degree of excellence of 
something." 

Oxford English Dictionary 


Standards and measures 
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Standards can help you figure out what, and 

sometimes how, you should measure 



The impetus of software quality 















n 

r 






Th 

e \ 

fa 

lue 

loI 

The 

J_C 

»o 

ftv 

Nc 

m 

D, 
























fro 

m 

th 

e c 

:us 

tc 

>rr 

ie 

r’j 




























pe 

rsc 

>e 

cri> 

/e, 

s 

h( 

DU 

d c 

Iri 

V€ 

5 t 

h( 






t 

















qu 

aTit 

■y 

rec 

lui 

re 


iei 

nt 

s. 













t 

















































_ 

|~ — ipB 















r ^ 
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Understand the value 





Understand the value 





Specifications to enable value 


Requirements 



•Customer driven 
requirements 
•Industry driven 
process requirements 
•Regulations 
•Explicit quality 
requirements 


Operational 

Constraints 


•Budget - Cost 
•Budget -Time 
•Budget - Resources 


Pre-existing 

solution(s) 

•Legacy software 
•COTS sotware 
•Libraries 
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Don't overcomplicate; focus on what is 
important to the customer 


Value is realized as capability 










Protecting value from risk 






r 

T 





r 





r 





R 

is 

ks 

ith 

at 

cc 

DU 

Id | 

pre 

av 

en 

it 

th 

ie 













1 










Cl 

JS 

to 

me 

art 

re 

irr 

re 

sail 

■ 

Zl 

ng 

l < 

a 
























d( 

3S 

in 

3d 

ca 

P* 

ab 

TTTf\ 

/ ir 

id 

ice 

at 

e 

the 








1 A U 



m 






■ 





Vc 

all 

je 

of 

th 

e 

sc 

)ftv 

var 

•e 

its 

>€ 

Hf 

IS 























at 

r 

sk. 
































-V — 
V 

Unr 

■ 














nitigated risks degrade delivered value 
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Per software module risk assessment 



Each module is scored 1-5 per factors below 


Impact Factors 


Likelihood Factors 

Performance 


Complexity 

Operational S/W Control 


Testability 

Human Safety 


Degree of Innovation 



Developer Characteristics 
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1 Likelihood 



Risk Level 2 
Risk Level 1 


[7-13] 

[0-6] 





Risk flows up to capabilities 











Capability 

Software Component 

Software Modules 

Launch to Orbit 

None 

None 

Approach to Target 

m 

Trajectory Control 

* 

Math 

GNC 

Attitude Control 

Navigation 

Attitude Control 

* 

Math 

GNC 

Attitude Control 

Navigation 

Maintain Flight 
Systems 

* 

Establish and Maintain Power 

* 

Battery 

Power 

Establish and Maintain Thermal 
Control 

Thermal 

* 

Perform Fault Detection 

FSW 

Establish and Maintain 
Communication 

* 

Telecom 

Downlink 

Uplink 

Command 

Telemetry 



Leverage the progress of others 











r 








r 




T 


m 

3 i 

s 

n< 

3 

re 

ai 

5C 

in 

tc 

r 

e| 

pe 

sat 























th 

e 

s; 

ar 

ne 

r 

ni 

ist 

:ake 


m 

al 

ce 

! S 

ure 

* 






















tc 

r 

nitic 

iat 

te 

k 

n< 

DV 

i/r 

i r 

isks 

V 

vit 

h 








w 















P' 

ro 

ve 

in 

ri 

si 

< i 

nitic 

3atior 











■ 
















te 

jchr 

IIC 

|U( 


5. 































-V — 
V 

Start 

■ 















j 

with known mitigations for known risks 
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Known risks: Enemies of Quality 


Unrealistic expectations 


Incorrect specification 




Bad coding practices and constructs 


Inadequate testing 


Example risk mitigation activities 


Building the right thing 


Building the thing well 

• Prototyping 


• Code generation 

• Simulation 


• Unit testing 

• Requirements tracing 


• Coding standards 

• Design reviews 


• Code reviews 



• Monte Carlo testing 



Confirmation of building the right 


Other Risk Mitigations 

thing well 


• Independent verification and validation 

• Static analysis 


• Experience and training 

• Integration testing 


• Continuous quality plan re-evaluation 

• Coverage testing 



• Code metrics 
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Each of these mitigation activities combats one 

or more of enemies of quality. 






Scope with the risk assesment 
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Risk mitigation oc 
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risk level 
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Example risk mitigation scoping 


Mi 

J Dynamic verification activities, medium rigor 
Static verification activities, high rigor 
Validation activities, high rigor 

✓ 

s 



Risk Level 2 = [7-13] 
Risk Level 1 = [0-6] 
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Static verification activities, medium rigor 
Validation activities, medium rigor 



U 


Validation activities, medium rigor 
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Each risk level dictates risk mitigation methods 
applied with specific levels of rigor. 
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An analogy for rigor 



Rigor of 

strategy 

application 


a 

a 


How many 
stones do I 
need to look 
under? 





Rigor applied to static code analysis 



Prove code safe 

Analyze third set of potential 
run-time errors 

Apply coding standard X checks 

Analyze unreachable branches 
Analyze second set of potential 
run-time errors 

Identify external interfaces and 
perform tainted data analysis 


Identify systematic run-time errors 
Analyze non-terminating constructs 
Analyze first set of potential run- 
time errors 


Rigor applied to static code analysis 


Assurance Task 

SQO Level 1 

SQO Level 2 

SQO Level 3 

SQO Level 4 

SQO Level 5 

SQO Level 6 

Develop Quality Plan (AT-1) 

X 

X 

X 

X 

X 

X 

Identify Software Build Information (AT-2) 

X 

X 

X 

X 

X 

X 

Identify Source Code Metrics (AT-3) 

X 

X 

X 

X 

X 

X 

Apply Standards Based Rules (AT-4) 

OPTIONAL 
per IV&V 
effort 

OPTIONAL per 
IV&V effort 

OPTIONAL 
per IV&V 
effort 

OPTIONAL 
per IV&V 
effort 

OPTIONAL per 
IV&V effort 

OPTIONAL per 
IV&V effort 

Identify Systematic Runtime Errors (AT-5) 


X 

X 

X 

X 

X 

Analyze Non Terminating Constructs (AT-6) 


X 

X 

X 

X 

X 

Analyze Unreachable Branches (AT-7) 



X 

X 

X 

X 

Identify External Interfaces (AT-8) 




X 

X 

X 

Analyze FirstSubsetof Potential Runtime 
Errors (AT-9) 




X 

X 

X 

Analyze Second Subset of Potential 
Runtime Errors (AT-10) 





X 

X 

Analyze Third Subset of Potential Runtime 
Errors (AT-11) 






X 

Prove Code Safe (AT-12) 






X (ta rgeted 
modules) 

Perform Tainted Data Analysis (AT-13) 





OPTIONAL or 
Required for 
Information 
Assurance/Secur 
ity Focused 
Analysis 

OPTIONAL or 
Required for 
Information 
Assurance/Securi 
ty Focused 
Analysis 

Perform Dataflow Analysis (AT-14) 






X 



Rigor applied to static code analysis 


Impact List 

Impact Definition 

Impact Level 

SQO Level 

MISRA or standards 
compliance 

Neither causes harm to the system nor a programmer mistake. These are simply good practices or generally 
accepted standards to follow. 

Trivial 

1,2, 3, 4, 5,6 

Deadlock 

Two or more threads are waiting for each other to finish causing the process to freeze. These are related to 
Semaphores, Mutexes, and Race conditions. 

Critical 

2, 3,4, 5, 6 

Memory leak 

Improper memory management. These involve improper or neglected deallocation or use of memory. 

Minor - Major 

2, 3,4, 5, 6 

System crash 

Impacts system/crew safety which could lead to Loss of vehicle, loss of mission, or loss of life. 

Critical 

2, 3, 4, 5,6 

Undefined behavior 

Code defects whose behavior is not specified under certain conditions. The behavior may vary depending on the 
implementation, environment, or semantics. Resulting behavior can range from benign to critical. 

Major - Critical 

2, 3,4, 5, 6 

Possible programmer 
mistake 

Does not cause any major or critical issues, but areas in code that may be worth a look to determine if code was 
intentional or not. 

Minor 

3,4, 5, 6 

Unexpected behavior 
or results 

Suspicious code that may negatively affect the behavior, code flow, or calculation result if the code was not 
programmed as intended. 

Minor - Major 

3,4, 5, 6 

Unreachable code 

Written code that will not be executed. These could either be commented out code or defensive code. Worth 
investigating to see if intentional. 

Minor - Major 

3,4,5, 6 

Data loss 

Chance to truncate data when assigning between objects, storing results of a calculation, or passing data as 
arguments, when the new storage type is smaller. 

Major 

4,5,6 

Data exposure 

Security vulnerability allowing supposedly inaccessible or private data to be modified by a malicious user. 

Minor - Major 

5,6 

Security 

Security vulnerabilities that do not overlap with another impact category. These include the use of unsafe 
functions, unverified or tainted inputs, or weaknesses prone to user exploitation. 

Minor - Major 

5,6 

Code cleanliness 

Good practices to observe near the completion of a project such as declaring objects as const or non-const when 
appropriate. 

Trivial 

6 

Performance 

Impacts system performance such as timing or memory usage. 

Minor 

6 

Portability or cause 
compile issues 

Code defects that may not be an issue on the current system but may not work if compiled on a different 
environment or if implementation was not well understood. 

Trivial 

6 

Readability and 
maintainability 

No impact on the system other than the possibility of confusion if code was shared/maintained by multiple 
developers or reused in another project without proper rationale included. 

Trivial 

6 

Redefined behavior 

Built-in commands or operators are overloaded or redefined to have new behavior. May cause confusion, however 
it is a non-issue if the system is well understood and documented. 

Trivial - Minor 

6 

Unused data 

Possible development oversight. A parameter, status or calculation result was not used, indicating there may have 
been an initial intent but forgotten. 

Minor 

6 



Putting it all together 


Known Risks 


Requirements 


Risk Assessment 
Criteria 


Possible risk 
mitigation activities 
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Propose 

Solution 


Y 


Risk 

Analysis 
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Pre-existing 

solution(s) 
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Software 
Quality Plan 
Creation 



Operational 

Constraints 


Risk mitigation 
adjustments 





Software quality plan reassesment 


^ 

Re-evaluate 
Software 
Quality Plan 




• Budget - Cost Change 

• Requirements - Scope 
Creep 

• Schedule Slip 

• New Risk Mitigation 
Strategies 
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Your quality plan is an evolving, living process. 
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What can IV&V provide? 






What can MathWorks provide? 



Contribute or ask questions 


NASA IV&V 


Dan Painter :: 

ioseph.d.painter@nasa.gov 



MathWorks 

Matt Rhodes :: 
matt.rhodes@mathworks.com 


